Papers with sparse MoE architecture
MoKA:Parameter Efficiency Fine-Tuning via Mixture of Kronecker Product Adaption (2025.coling-main)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is one of the most popular PEFT methods . low-rank update mechanism of LoRA somewhat limits its ability to approximate full-parameter fine-tuning during training process. |
| Approach: | They propose a parameter-efficient fine-tuning framework that combines Kronecker product with the Mixture-of-Experts method to achieve parameter efficiency and better model performance. |
| Outcome: | The proposed framework outperforms existing methods on the GLUE benchmark and instruction tuning tasks for large language models. |
CLIP-UP: A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcycling (2025.findings-emnlp)
Copied to clipboard
Xinze Wang, Chen Chen, Yinfei Yang, Hong-You Chen, Bowen Zhang, Aditya Pal, Xiangxin Zhu, Xianzhi Du
| Challenge: | Mixture-of-Experts (MoE) models are crucial for scaling model capacity while controlling inference costs. |
| Approach: | They propose an alternative training strategy that converts a dense CLIP model into a sparse MoE architecture. |
| Outcome: | The proposed training strategy outperforms dense models on COCO and Flickr30k benchmarks. |
GMoE: Global Mixture of Experts with Logit Propagation (2026.acl-long)
Copied to clipboard
| Challenge: | Sparse Mixture of Experts architectures retain large memory footprints and exhibit significant redundancy, both within and across layers. |
| Approach: | They propose a sparse mixture of experts architecture that uses global experts shared across all layers and adds a Local Expert per layer for layer-specific adaptation. |
| Outcome: | The proposed architecture reduces computational cost by activating only a subset of experts per token while maintaining strong performance. |